Modern camera systems rely heavily on computational processing to produce high-quality digital images. Even with relatively simple camera designs, reconstructing a photograph from raw sensor measurements is a complicated process that involves many complex tasks, including dead-pixel elimination, noise removal, spatial upsampling of subsampled color information (e.g., demosaicking of Bayer color filter arrays), sharpening and image compression. More specialized camera architectures may require additional processing.
FIG. 1 is an illustration of an exemplary conventional hardware implemented image processing pipeline on an image signal processor (ISP) for photograph reconstruction. The complexity of the process for reconstructing a final output photograph 20 from raw sensor measurements 10 is typically tackled by splitting the image processing into several independent stages forming a reconstruction pipeline as shown in FIG. 1. Conventional image processing pipelines for capturing, displaying, and storing images are usually defined as a series of cascaded modules, e.g., denoising 30, demosaicking 40, etc., each one responsible for addressing a particular problem and working on an output of an upstream module. Splitting image reconstruction into smaller, seemingly independent tasks has the potential benefit of making the whole process more manageable, but this approach also have several severe shortcomings
First, most of the individual stages are mathematically ill-posed and rely heavily on heuristics and prior information to produce good results. The following stages then treat the results of these heuristics as ground-truth input, aggregating the mistakes through the pipeline. In other words, while the pipeline approach offers many of the benefits of the classic divide-and-conquer methodology, it also introduces a cumulative error, as each step in the pipeline only considers the output of the previous step, not the raw sensor data. The results of this cumulative error include artifacts such as noise residual, zipper artifacts, or color fringing, etc.
Secondly, the individual stages of the pipeline are in fact not truly independent, and there often exists no natural order in which the stages should be processed. For example, if denoising 30 follows demosaicking 40 in the pipeline, the demosaicking step must be able to deal with noisy input data when performing edge detection and other such tasks required for upsampling of the color channels. All these challenges of the pipeline approach are exacerbated by more complex image processing needs in modern computational cameras.
Further, another problem with current ISPs is that their architecture is rigid and, therefore, implementing new computational photography applications is not straightforward or efficient. Because the ISP pipeline is implemented in hardware and is specific to one design it is not easily portable to different hardware implementations, e.g., different camera models, etc. For example, several new photography applications and camera types have been developed that deviate from a regular Bayer image capture and, therefore, map poorly to the rigid design of current ISPs.